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ABSTRACT 



The types and frequency of markings students made on 



multiple choice tests were studied as an indication of students' 
metacognitive test-taking strategies. Also studied was whether instruction in 
test-taking strategies affected students' fluency and flexibility of strategy 
use as demonstrated through the marks they made on tests. Participants were 
90 undergraduates in 16 sections of a learning strategies course. Students 
were tested on three occasions in the semester. Answer indication and option 
elimination marks were most commonly made, with option elimination marks most 
commonly associated with high test scores before test-taking strategy 
instruction. After strategy instruction, and perhaps as a result of 
increasing the number of markings students made overall, none of the 
categories of markings were significantly correlated with high test scores. 
Following test-taking strategy instruction, both the fluency and the 
flexibility of types of markings increased, although flexibility decreased 
when students were tested several weeks after instruction. There was a 
difference in the pattern of markings of successful versus unsuccessful test 
takers. Option elimination, answer change, and selective item markings were 
evidently used by the student as metacognitive aids. (Contains 10 tables and 
22 references.) (SLD) 
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Introduction 

Multiple choice tests are one of the most common ways we have of evaluating 
student achievement at the college level. Research on test-taking strategies suggests 
that cognitive strategies may be successfully instructed to and applied by college 
undergraduate students on multiple choice tests (Dolly & Williams, 1986). In studies 
comparing academically successful to academically unsuccessful students, it has been 
found that high performers demonstrate a greater knowledge of effective test-taking 
skills and use learning strategies that lead to a deep, rather than to a superficial level 
of encoding and (Bruch, Pearl & Giordano, 1986). Weinstein (1988) suggests that 
instruction of these strategies within a metacognitive framework will promote transfer 
and application of these strategies across different academic situations. The length of 
time spent on instructing test-taking strategies also seems to be of importance: in a 
1985 meta-analysis of 24 programs which taught test- taking skills to elementary and 
secondary school students, Samson found that training programs which lasted five 
weeks or longer produced significantly greater results than did shorter progrems. In 
addition, Dolly and Williams (1986) found that, while testwiseness strategies may be 
taught, these strategies have limited generalizability to other exams. Their results 
show that these strategies are effective only when applied to items which are 
susceptible to test wiseness strategies. 

Testwiseness 

Previously, research on students' responses to multiple choice questions has 
come from the field of item response theory and from the analysis of individual test 
items. These techniques allow us to evaluate the quality and difficulty of items on 
exams along with giving us a glimpse into what item alternatives or concepts are most 
difficult for students. Testwiseness or strategic test-taking encompasses a slightly 
tangential, if related fields. 
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The idea of "test wiseness" was originally discussed by Thorndike (1951 ) and 
thought to be a variable which could possibly affect test reliability. Thorndike 
considered testwiseness to be a general and lasting cognitive factor in that the manner 
in which an individual responded to tests affected her scores across content areas. 

Given this view, testwiseness can be seen as part of any test score. However, Thorndike 
considered testwiseness to be part of the ertvrin an individual's test score. Currently, 
researchers in the area of testwiseness have differing views. Scruggs & Lifson (1985) 
argue that test-wiseness is a large source of variance that is commonly found in tests 
and that it is not related to general intelligence, stating "the influence of test-wiseness 
has been greatly overestimated." Conversely, Green & Steward (1984) see test wiseness 
as simply an artifact of one general cognitive ability. They view it as a highly 
developed reasoning ability which is combined with both general and specific 
experience. Other investigators (Dolly & Williams, 1986; Evans,1984) believe that test- 
wiseness is not a general ability, but that it is cue specific given the nature of 
individual items. 

Weinstein (1988) uses the term “test-taking strategies" to refer to the concept of 
test wiseness. As in the definition of test wiseness, an individual who employs test- 
taking strategies is expected to get a higher score on a test than an equally able 
individual who does not employ test-taking strategies. Earr, Pritchard & Smitten (1990) 
have found that students approach a test in three different ways; by employing reading 
strategies, by using an overall approach to the test task and by using test-taking 
strategies. However, few investigations indicate what kind of strategies are 
significantly related to increased test performance. Research from the field of reading 
(Anderson & Armbruster, 1984; Hist & Kirby, 1989) suggests that underlining and 
annotations may facilitate comprehension while reading test questions. Other 
investigators have found that changing answers (Hanna, 1989); a low level of anxiety 
(Covington & Omelich, 1987); and using an outline before studying (Mannes & Kintsch, 
1987); may aid test performance. In a study of the type of test markings that college 
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students made on a multiple choice exam, Kim and Goetz (1991 ) found that item 
elimination marks are significantly correlated with high test scores. In general, 
results from the metacognitive literature suggests that learners that proactively 
process information, such as test items, are more likely to understand and recall what 
they learn. 

Metacoenition and Strategy Use 

Strategic learning, higher-order metacognitive processes, and executive 
mechanisms ere critical components in successful learning and transfer (Borkowski & 
Kurtz, 1987; Gamer & Alexander, 1989). However, superordinate control processes have 
been interpreted in different manners by different authors. Flavell (1979) defines 
metacognition as "knowledge that takes as its object or regulates any aspect of any 
cognitive endeavor." In actuality, there are two parts to this definition; knowledge 
about one's own cognition that is statable or accessible to the individual and knowledge 
about the regulation and control of this knowledge (Campione, 1987). A further 
definition of these terms and how these components might function together in 
processing test items is necessary. 

Erecuti rap Control 

Executive control is the regulation and control of one's knowledge. It involves 
high-level management of learning. Self-regulatory activities are engaged during an 
attempt to learn or solve a problem and involve planning, selecting, implementing, 
monitoring, modifying and evaluating (Weinstein, Meyer & Stone, 1991). Research in 
the late 1970's and early 1980's began to include an executive control component, with 
the result that transfer of the trained strategies was enhanced. At this point, 
researchers involved in courses on learning-to-learn began incorporating this 
element of metacognition into their work (Campione, 1987). 
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A rareness of cognition 

Metacognition includes the component of awareness of one's cognition and 
cognitive resources. In the early 1980’s the typical training study typically involve the 
students as active participants in the training process. Subjects were asked to use a 
sequence of steps in problem solving but were not informed why or under what 
circumstances such a strategy should be used. Such studies tended to produce limited 
transfer. 

Strategies 

Strategies ere higher-level plans that help the learner to identify the specific 
skills needed for a learning task (Pressley, Borkowski & O'Sullivan, 1984; Veinstein & 
Underwood, 1983). A large number of studies have attempted to instruct individuals in a 
specific strategy and then to measure transfer on a novel task. The results of these 
studies have been that individuals could be taught to carry out strategies, which 
resulted in improvement in performance but they frequently abandoned the strategy 
when the experimenter ceased prompting its use; and then failed to apply the strategies 
to new problems where they would be appropriate. Strategy training has been 
supplemented with metacognitive training which include the awareness and executive 
control components discussed previously. 

Metacognition and Test-taking 

A student who is metacognitively aware monitors her comprehension. A major 
function of executive control is to aid when comprehension failure occurs. In the case 
of test-taking, the student must monitor the items she is answering, be aware of 
especially problematic items, monitor what knowledge she has which can be applied to 
an item, and which items she should skip and work on later. When a strategy is not 
successful, a different one must be selected in order to answer the item. This process of 
executive control, of modifying one's strategies, is dependent upon the fluency and 
flexibility of strategies which are employed. Veinstein (1988) describes fluency as the 
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amount of strategies which one possesses, while flexibility refers to the variety of 
strategies one possesses and can apply. She sees fluency in strategy use as aiding in 
the access of /issfetetfportions of existing knowledge, vhile the flexibility of strategy 
use aids in accessing ofrWfefisw/portions of prior knowledge. Flexibility in strategy use 
seems to be the primary problem which most students have in learning and studying. 
For example, if a student has a comprehension problem while attempting to answer a 
difficult item, she might try rereading the item. If this does not help her, then she is 
likely to skip the item, rather than implementing another strategy. This might be an 
effective test-taking strategy initially, but eventually, if the student is to answer the 
item correctly, she must employ a successful strategy (rather than simply guessing) to 
answer an item correctly. 

Test-taking Strategies and Test-markings 

Kim and Goetz (1991 ) have investigated the types of markings which college 
students make on multiple choice tests. They argue that successful students might use 
more sophisticated test-taking strategies than do less successful students. In their 
study, they examined six types of test-markings which students made in response to 
multiple choice items; answer indications, option eliminations, key terms, selective 
item markings, elaborations and answer changes. They found that option elimination, 
key term, selective item, elaboration, and answer change markings were all 
significantly related to item difficulty. In each of these cases, the use of the strategy 
increased as the item difficulty increased. In addition, option elimination marks were 
found to be positively correlated with students' total test scores. Thus, item markings 
seem to be strategic in nature in that they adapt and are modified, given the 
requirements of the item. 

Test markings might be an effective way to measure students' metacognitive 
test-taking strategies. The firequencyx^ such markings can be utilized as a measure of 
the fluency of strategies vhich students employ to attempt to answer items correctly. 
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The vsnefyot markings used can likewise be utilized as a measure of the flexibility of 
the strategies used on the items. 

For this study the questions of interest are: What type of test markings are the 
most commonly used by students taking a multiple choice test? Are certain kinds of 
markings correlated with higher test scores? With correctly answering an item? With 
incorrectly answering an item? As an item becomes more difficult, does the student 
increase the fluency and flexibility of the strategies which she employs? In addition, 
does the instruction of test-taking strategies, within a course on metacognitive strategy 
use, affect the students' fluency and flexibility of strategy use? 

Method 

Sub/ecrs. 

397 undergraduate students enrolled in 16 sections of a learning strategies 
course at a major Southwestern university were given, as part of the requirements of 
the course, three tests over the course of the semester. Of those students, 90 were 
randomly selected to participate in this study. This student sample was stratified so that 
the students came from one of four different sections of the course, each with a 
different instructor. Instructors varied in the amount of previous teaching experience 
that they had; one instructor had previously taught the course for six semesters, one 
instructor had four semesters of experience, one had three semesters of experience and 
one had not previously taught the course. 

Although the subjects came from sections taught by four different instructors, 
the exam was based on common course material and common texts. Of the subjects, 12 
percent were seniors, 19 percent were juniors, 35 percent were sophomores and 34 
percent were freshmen. Subjects from the four sections who did not take all three tests 
over the duration of the semester were dropped from the sample. 
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Materials. 

The first test was administered six weeks after the beginning of the spring 
semester. It contained 13 multiple choice questions and two short answer items. The 
test covered course material on cognitive learning theory and time management. It 
counted for 15% of the final course grade. 

The second test was administered eleven weeks into the semester. It also 
contained 15 multiple choice questions and two short answer items, but also included a 
short essay. The test covered course material on cognitive learning theory, strategy 
use, reading comprehension, notetaking and test-taking. The test-taking unit included 
a 50 minute lecture on how to prepare for and take tests. Test-taking strategies which 
were presented included; test-preparation strategies, time-management strategies, test- 
taking strategies before a test, during and after a test. Strategies for use when taking a 
multiple-choice exam were explicitly taught using multiple-choice questions as models. 
Types of test-markings were not categori 2 ed or defined for students, but were modeled 
during instruction. Test *2 was administered two days after the students had received 
instruction on test-taking strategies. This second test counted for 20% of the total 
course grade. 

The final test was administered 15 or 16 weeks after the beginning of the 
semester, at the end of the course. It contained 45 multiple-choice items. The students 
had received additional instruction on stress-management and on integration of the 
above material. The final test counted for 22% of the total course grade. 

Procedure 

Students were tested in the rooms in which they received their lectures and the 
test was administered by their regular instructor. They received scantrons, which 
were later machine scored, along with their test booklets. They were verbally 
instructed to bubble in the test answers on the scantron card, but were told that they 
could write or mark on the booklets as they wished. These instructions were also 
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written on the front of the test booklet. Both the scantrons and the test booklets were 
collected at the end of the testing period. 

Test booklets were examined and the categories used by Kim and Goetz (1991) 
were used to code the markings of each item; 1 ) answer indication markings (ie„ 
circling, checking, writing or otherwise indicating the response option to be marked 
on the answer sheet), 2) option elimination marks (ie., crossing or rubbing out 
response options), 3) key term marks (ie„ circling, underlining or drawing a box 
around the key terms in the stems or options of the item), 4) selective item marks (ie., a 
mark such as a question mark, a star or a check beside a subset of items), 3) elaboration 
or annotation marks (ie„ additional words, drawings or diagrams written to the side of 
an item), and 6) answer change marks (ie., erasing or crossing out of an answer 
indication mark). A seventh category was added, that of "other" to account for marks 
which did not fall into the above categories, or for marks that were judged ambiguous 
by the raters. 

Items from Test # 1 were used to calculate interjudge agreement on the coding of 
markings of the items. A total of 1080 items from 72 different subjects were coded by 
two judges. The codings from these two judges were then compared (see Table 1 ). The 
coding of the items had extremely high interjudge agreement, with a8 index of .99. 

A total of 1350 items were examined and coded (90 subjects X 15 items) for both 
Tests *1 and * 2 . For Test # 3, 4050 items were examined and coded (90 subjects X 45 
items). Each item had the possibility of being coded as having no marks at all or 
having any combination of up to all of the seven types of markings. This data was 
analyzed as to the overall frequency of each type of markings and the frequency of 
markings for each item. Point biserial correlations were calculated in order to examine 
the relationship between the types of markings a student used on a given item and the 
total test score received by the student. 

The scantron cards were used to calculate the students' total grade on the exam 
and to determine the class average and standard deviation for each exam (Test 1 : 
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Mean=ll.l (74%), SIM. 8, Test 2: Mean=13.0 (86%), SIM .5; Test 3: Mean® 33.05 (73%), 
SD=4.8>. Items vere additionally analyzed to determine the difficulty level and the 
discrimination level of each item. In order to control for the differences in the 
difficulty level of each test as a whole, four items were selected from each test. Using 
the previously mentioned measures, these four items vere equated on their difficulty 
levels and discrimination levels and selected from each exam. For each exam, a very 
easyitem (p=.94-.95, R(IT)=.24-.38); an easy item (p=.80-.83, R(IT)® .32- .44); a moderately 
difficult item (p=.62- .67, R(IT)=.40-.48); and a difficulty item (p=.43-.46, R(IT)=.41-.46) 
vere selected. The types of markings from each of these items vere totaled for each 
subject to give a measure of the flexibility of types of test markings used. Thus, each 
level of item difficulty had a total number of 90 observations (one per subject). 

Results 

The number of students marking their test increased over the course of the 
semester. In Test 1, 80 of the students marked their test* in Test 2, after test-taking 
strategy instruction, 84 marked their tests and in Test 3, 87 marked their tests. Overall, 
92.9% of the subjects made some category of marking on their test across the three 
exams. 

The number and percentage of items vhich exhibited one of the seven 
categories of test marks are shown in Tables 2-4. For each test, answer indication 
marks vere the most common type of marked used, followed by option elimination 
marks. For the first two tests, elaboration marks vere the most third common marking, 
while for the final, key term markings vere the third most common type of markings. 
The key term markings, "other" markings and elaboration markings vere the fourth 
most popular type of markings for Tests # 1, # 2 and # 3, respectively, while the fifth 
most commonly used marks vere "other" for both Tests # 1 . key term for Test * 2 , and 
selective item marks for Test # 3. The sixth most popular marking was the selective item 
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marking for Tests *\ and *2. while for Test # 3 it was the "other" category of marking. 
For all three tests the least common type of marking were answer change markings. 

In Test *2. after test-taking strategy instruction, the mean occurrence of all 
types of markings increased (see Tables 2-4), with the exception of key term marks and 
elaboration marks. In Test # 3, all types of markings, with the exception of elaboration 
marks were increased over the relative markings from Test # 1 . The mean occurrence 
of all types of markings continued to increase in Test # 3 over those in Test *2, with the 
exception of answer indication and option elimination markings (the most frequent 
type of marking) and markings classified as "other." The frequency, or fluency, of test 
markings did increase from Test # 1 to Test # 3 over the semester, following test-taking 
strategy instruction. 

Correlational analysis was used to determine which of the types of test markings 
were associated with overall high test scores. Frequency of test markings were summed 
across all 15 items in Tests *1 and *2 and across all 45 items in Test # 3. Correlations 
between test scores and types of markings are shown in Tables *5, *6 and * 7 . For Test 
*\, only option elimination markings were significantly (p<_ 05) positively correlated 

with high test scores, while option elimination markings were negatively correlated 
with high test scores. This indicates that those with higher test scores tended to use 
option elimination test scores, while those scoring lower on the test tended to use 
answer indication markings. 

On Test *2. after test-taking strategy instruction, none of the types of markings 
were positively correlated with higher test scores. This implies that both successful 
and unsuccessful students were using test-marking strategies previously used only by 
the successful test-takers. In test # 3, five to six weeks after the test-taking strategy 
instruction, answer indication, option elimination, key term, and selective item marks 
were positively correlated with a high test score, while "other" marks were negatively 
correlated with high test marks. Taken together, these results suggest that option 
elimination marks tend to be associated with high test scores, in both those who are 
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testwise and those who are not testwise, but not immediately following test-taking 
strategy instruction . 

Some of the categories of markings were correlated with the difficulty of the 
items (see Tables *5-7). In Test *1, key term, selective item, elaboration and answer 
change marks were associated with difficult items. In Test *2, selective item and 
answer change markings were also associated with difficult items, as were elimination 
markings. Difficult items on Test *3 were significantly correlated with the same types 
of markings as were those on Test # 1 : key term, selective item, elaboration and answer 
change markings. Thus, it seems that the frequency of selective item marking and 
answer changes are consistently associated with difficult items. 

Also of interest, is if certain markings are correlated with correctly or 
incorrectly answering an item. In order to control for the difference in the difficulty 
level of each item, four items from each test were equated as described earlier. Phi 
correlations were calculated for each item to determine association between types of 
markings and correctness of the item. In Test *1, option elimination markings were 
positively correlated with answering all but the difficult items correctly. In Test *2, 
after test-taking strategy instruction, selective item and "other" marks were associated 
with correctly answering easy items, while answer identification markings were 
positively associated with answering difficult items correctly. In Test *3, depending on 
the difficulty level of the item, elaboration, option elimination, and answer change 
markings were associated with correctly answering an item. Again, option elimination 
markings seem to be fairly consistently associated with correctly answering items, 
except directly after test-taking strategy instruction. 

Conversely, several types of markings were negative correlated with getting an 
item correct. In Test *1, answer indication markings were associated with incorrectly 
answering an easy, moderate or difficult item. In Test # 2, answer changes were 
associated with incorrectly answering very easy or moderate items. In Test *3, answer 
indication marks were, as in Test *1, associated with incorrectly answering difficult 
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items. Answer indication marks seem to be consistently associated with answering an 
item incorrectly before test-taking strategy instruction. 

Since the ordering of the types of markings, along with the mean number of 
occurrences of markings, seemed to change from test to test, further analysis was 
conducted to examine the changes in flexibility of merkings from test to test. 

As a measure of flexibility of marking strategies, the total number of types of 
markings for the four equated items from each test were analyzed using log-linear 
analysis. The category of "7 different types of markings", as suggested by Vickens 
(1989,p.l3)) was eliminated since none of the three tables contained data in this 
category. Goodness of fit was calculate for the null hypothesis of “no change over 
time" using a Markov chain analysis for each of the item difficulty levels. This allowed 
the change in the number of types of marking of each student to be analyzed across the 
three tests. In the case of a nonsignificant change in numbers of types of merkings, a 
test of marginal homogeneity was run to determine if there was a significant change 
in the number of types of markings from Test # 1 to Test * 2 . 

For all item difficulty levels, the number of types of markings increased after 
test-taking strategy instruction from Test # 1 to Test *2 (see Table # 10). None of the 
types of item markings increased from Test *2 to Test # 3, and, in some cases, the types 
of markings dropped. Overall, however, the number of types of marking increased 
over the semester in response to very easy items and moderate difficulty items. Types 
of markings did not significantly change over the three tests for easy and difficult 
items. These results suggest that flexibility of test-marking strategies increase 
immediately after test- taking strategy instruction, but do not continue to increase, and, 
in the case of easy and difficult items, drop back to their original level. 
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Discussion 

In this study several questions were of interest. The first question was: what 
type of test markings are commonly used by students on a multiple-choice test? Results 
of this study concur with the findings of Kim and Goetz (1991 ) in that answer indication 
and option elimination marks are most common. Also in concordance with their 
findings, was that option elimination marks are most commonly associated with high 
test scores before test- taking strategy instruction. However, immediately after test- 
taking strategy, and perhaps as a result of increasing the number of markings that 
students make overall, none of the categories of markings were significantly 
correlated with high test scores. Several weeks after instruction, option elimination 
marks, along with four other categories of markings are significantly correlated with 
high test scores. Since the number of these categories of markings are increasing 
from test to test, we might hypothesize that some students have begun to use test 
marking strategies that differ from their less successful peers. Two reasons could 
explain this change: the additional instruction of the learning strategies course is 
helping students become more effective in their strategy usage, or, since the Test # 3 
was the last exam of the course, successful students tend to use different and more 
effective strategies. 

Certain types of item markings are correlated with correctly answering an item. 
Before strategy instruction, option elimination, at least for the 3 easiest categories of 
items, is most effective. On Test *3 it is also an effective strategy for easy and moderate 
items. For items of moderate and difficult levels in Test *3, answer change strategies 
were associated with correctly answering an item. This suggests that a student's score 
will be raised only after test-taking strategy instruction couched within a learning 
strategies course: Answer change was negatively correlated with answering an item 
correctly before and immediately following test-taking strategy instruction. 

Students take into account the difficulty level of an item when they make test 
markings. Answer change and selective item markings consistently are applied by 
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them. This implies that a student is likely to mark end item to return to later if it is 
difficult, and she is more likely to change the answer to the item. However, making 
answer changes does not appear to be helpful, unless the item is of a moderate or 
difficult level, and she has continued to receive instruction in a learning strategies 
course. 

Does test-taking strategy instruction appear to affect the flexibility of student 
test-marking strategies? It appears from this analysis that there was a significant 
difference between the flexibility of test-taking strategies across the three exams. The 
flexibility of the types of markings that are made in response to a difficult item 
increase immediately after test-taking instruction. This flexibility, however, then 
decreases after several weeks have passed. 

In summary, following test-taking strategy instruction, both the fluency and 
the flexibility of the types of markings increases. The flexibility of the types of 
markings decreased, however, when the students were tested several weeks after the 
instruction. This is a finding repeatedly manifested in research on strategy 
instruction. Near transfer and application of a strategy is common, far transfer is 
elusive. In addition, flexibility of strategies is an area in which students demonstrate 
particular deficiencies (Weinstein, 1988). It should be noted, however, that the fluency 
or frequency of the types of markings continued to increase across the three types of 
the tests. This finding suggests that, while the types of marking strategies did not 
increase, within the individual categories, markings did increased, even weeks after 
strategy instruction. Continued enrollment in a learning strategies course appears to 
affect test marking. 

Test marking patterns are correlated with a number of factors and seem to be 
strategic in nature. There is a difference in the patterns of markings of successful 
versus unsuccessful test takers. These markings are affected and increased by test- 
taking strategy instruction. Several of the types of markings (option elimination, 
answer change, selective item) are implemented by the student to metacognitively aid 
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her in answering items of certain difficulty levels correctly. Finally, the instruction 
of test-taking strategies within a learning strategies course increases the frequency of 
these strategic markings. 
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Table 1: Log-linear test of Interjudge Agreement: Based on codings of 
responses of 72 students on 15 items (1080 items total) from 
Test # 1. 



JUDGE # 2 



JUDGE #1 



ANSIND 


ELIM 


KEY TERM 


SELECT 


ELAB 


ANSCHNG 


OTHER 


MISSING 


ANSIND 


1004 


5 


1 


0 


0 


0 


2 


6 


EL1M 


6 


1004 


0 


0 


0 


0 


5 


10 


KEY TERM 


2 


1 


1004 


0 


0 


0 


0 


0 


SELECT 


0 


0 


0 


1004 


0 


0 


1 


1 


ELAB 


0 


0 


0 


0 


1004 


1 


2 


6 


ANSCHNG 


0 


0 


0 


0 


0 


1004 


1 


0 


OTHER 


2 


1 


0 


0 


1 


1 


1004 


2 


MISSING 


4 


3 


1 


2 


4 


4 


2 


1004 



The "missing category" refers to cases in vhich a category marked by one of the judges 
vas not marked by the other judge. 



Interjudge Agreement: 

2 

Data Source Model G 

Erom Table *1 No AXB interaction 32706.4077 

Table *1: Diagonal cells 
replaced -with structural 

zeros No AXB interaction 49.6501 



2 2 

Index 6= G 1-G2 = 32706.4077-49.6501 = .99 
2 | 49.6501 

G 1 




The relative proportion of the structure of Table *1 that depends on actual agreements 
is 99 percent. Thus, 99 percent of the structure in Table # 1 is due to the agreements of 
the judges. 
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Table 2: Categories of Markings and Percentages of Total Markings Ranked in Order of 1 8 
Frequency of use by Students on Test *1. 



Category Percentage 



Answer Indication 


55.0 


Option Elimination 


27.6 


Elaboration 


8.6 


Key Term 


6.1 


Other 


4.4 


Selective Item 


3.5 


Answer Chanee 


1.0 


N= 1350 items 




Table 3: Categories of Markings and Percentages of Total Markings Ranked in Orde 


the Frequency of Use by Students on Test *2. 


Category 


Percentage 


Answer Indication 


71.9 


Option Elimination 


43.0 


Elaboration 


5.9 


Other 


5.8 


Key Term 


5.2 


Selective Item 


4.7 


Answer Change 


ZSi 



N=1350 items 

Table 4: Categories of Markings and Percentages of Total Markings Ranked in Order of 
the Frequency of Use by Students on Test 



Category Percentage 



Answer Indication 


68.4 


Option Elimination 


40.8 


Key Term 


7.4 


Elaboration 


62 


Selective Item 


5.9 


Other 


5.3 


Answer Change 


2.6 



o 

ERIC 



N=1350 
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Table 5: Intercorrelations among the Seven Marking Categories, Total Test 
Score and Difficulty Level for Test *1 
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Table 6: Intercorrelations among the Seven Marking Categories, Total Test 
Score and Difficulty Level for Test *2 
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Table 7: Intercorrelations among the Seven Marking Categories, Total Test 2 1 

Score and Difficulty Level for Test *3 
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Test-Taking Strategies 



Table 8: Types of Markings Correlated with Correctly Answering an Item 22 
(Pi.05. n=90) 



Difficulty Level 






of Item 


TEST #1 


TEST * 2 


TEST #3 


Very Easy 


Option Elimination 


None 


Elaboration 


Easy 


Option Elimination 


Selective item 


Answer indication 
Option elimination 
Elaboration 


Moderate 


Option Elimination 


None 


Option elimination 
Selective item 








Answer change 


Difficult 


None 


Answer identification 


Option elimination 
Answer change 



Table 9: Types of Markings Correlated with Incorrectly Answering an Item 
(p<.05, n=90) 



Difficulty Level 
of Item TEST # 1 


TEST #2 


TEST #3 


Very Easy 


None 


Elaboration 
Answer Change 


None 


Easy 


Answer indication 
Elaboration 
Answer Change 


None 


None 


Moderate 


Answer indication 


Answer change 


Key term 


Difficult 


Answer indication 


None 


Answer indication 
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Table 10: Log-Linear Analysis of the Increase in Types of Test Markings 
made from Tests # l- # 3 using Markov Chains* 



23 



Difficulty Level Marg.Homo. Likelihood 



of Item 


Tests *1-2 


df 


Increase? 


Tests *1-3 


df 


Increase? 


Very easy 


31.9758 


10 


Yes 


48.1986 


30 


Yes 


Easy 


45.3453 


10 


Yes 


25.3942 


30 


No 


Moderate 


14.3258 


6 


Yes 


31.667 


20 


Yes 


Difficult 


21.3234 


15 


Yes 


18.4518 


42 


No 



n=90 Subjects per item type 
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